Skip to content

SEP-1864: Remediate HIGH/CRITICAL CVEs in the side-car image - #1373

Merged
yyyyyyyan merged 1 commit into
mainfrom
SEP-1864
Aug 19, 2026
Merged

SEP-1864: Remediate HIGH/CRITICAL CVEs in the side-car image#1373
yyyyyyyan merged 1 commit into
mainfrom
SEP-1864

Conversation

@yyyyyyyan

@yyyyyyyan yyyyyyyan commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Takes the published SEP side-car image from 101 HIGH/CRITICAL Trivy findings to 17, and dispositions each of the remaining 17 as a documented, purl-scoped exception. Three files change; no application code is touched.

sidecar/Containerfile.sidecar — three edits to the final stage

Edit Effect
Base pin python:3.11.14-slimpython:3.11.16-slim Debian 13.3 → 13.6, bringing fixed openssl (3.5.6-1~deb13u2), libcap2, and wheel 0.46.3
Delete the first apt-get block entirely Removes g++ (which pulled 43 transitive packages including linux-libc-dev) and the WeasyPrint stack (libpango-1.0-0, libpangoft2-1.0-0, fontconfig, fonts-dejavu, shared-mime-info, which brought libglib2.0-0t64 and libxml2). netcat-openbsd is installed separately further down and stays
Add apt-get upgrade -y to the surviving layer The pin bump alone leaves util-linux 2.41-5; Debian published 2.41.5-0+deb13u1 after the tag was cut. update → upgrade → install → clean stay in one RUN so the upgrade never runs against a stale index

RUN pip install --no-cache-dir wheel is also dropped. It was a no-op: python:3.11.14-slim already shipped wheel 0.45.1, so the line resolved as already-satisfied and installed nothing. The bumped base ships 0.46.3, past the 0.46.2 fix, so the base bump is what clears that finding.

.trivyignore.yaml (new, repository root) — 12 entries covering the 17 residual findings, each carrying id, purls, statement and expired_at: 2027-02-19. Every statement gives a reachability argument plus the condition under which the entry should be removed, not merely "no upstream fix".

changelog.d/SEP-1864.security.md (new) — security fragment.

Two things a reviewer should know

1. The exception file is inert unless the scan passes --ignorefile. Trivy's DefaultIgnoreFile is .trivyignore, so a bare trivy image never loads the YAML form — it parses the full schema only for a path handed to it explicitly. PMM's own scan (percona/percona-docker/.github/workflows/trivy_scan_pmm.yml) passes no such flag. The requirement is stated in the file's own header comment so the follow-up CI gate inherits it rather than rediscovering it.

2. purls on every entry is load-bearing, not decoration. In Trivy's pkg/result/ignore.go, matchPURL() returns true when the list is empty, so an unscoped entry suppresses its CVE for every package — including one where the reachability argument does not hold. Every justification here is package-specific ("nothing executes perl", "SEP serves no QUIC", "the archive paths link zlib rather than invoking the gzip binary"), so each entry is scoped to exactly the packages it assessed. The constraints carry no version, so a Debian point release does not silently un-except a finding.

setuptools: why no upgrade line

An earlier iteration added RUN pip install --no-cache-dir --upgrade "setuptools>=81,<82" to clear the jaraco.context 5.3.0 finding while preserving pkg_resources. It was measured to be inert and removed:

  • requirements.txt pins setuptools==83.0.0. The builder stage bakes setuptools-83.0.0-py3-none-any.whl into /wheels, and the next recipe line — RUN pip install --no-cache-dir /wheels/*, untouched here — reinstalls it. Any bound placed above that line is overwritten.
  • pkg_resources is consequently already absent from the pre-change image, through that same unchanged line. Verified directly: installing setuptools 83.0.0 on the old base python:3.11.14-slim removes it. Not a regression introduced here.
  • The security goal is met regardless: setuptools 83.0.0 vendors jaraco_context-6.1.0, and the built image contains no stale 5.3.0 copy anywhere under /usr/local/lib/python3.11.

Restoring a bound would mean downgrading against the lockfile. Constraining setuptools repo-wide is a separate decision.

Verification performed

Live evidence against the rebuilt image. Trivy pinned to v0.69.3, the version PMM installs, run as docker.io/aquasec/trivy:0.69.3 against a podman save --format docker-archive export of sep:HEAD.

Check Result
Image builds with no compiler present make builder and RELEASE_VER=HEAD make image both exit 0
g++ and the WeasyPrint stack gone g++ absent; linux-libc-dev, libglib2.0-0t64 and all five rendering libs not-installed; libxml2 reports unknown ok not-installed
Base layer refreshed and upgraded /etc/debian_version 13.6; util-linux 2.41.5-0+deb13u1; openssl 3.5.6-1~deb13u2; libcap2 1:2.75-10+deb13u1+b1
Python packaging state setuptools 83.0.0, wheel 0.46.3; only jaraco_context-6.1.0.dist-info present
Container healthy healthcheck.sh printed healthy, exit 0. All five non-one-shot supervisord programs RUNNING; all three migrations complete with their sentinels
App set unchanged verified restricted: 5 app packages (atw, framework, inventory, mysql_backups, shared), exit 0
Bare rescan 17 HIGH/CRITICAL, all in the Debian layer; 0 python-package findings
Gated rescan --ignorefile .trivyignore.yaml --exit-code 1 → 0 findings, exit 0
Reconciliation 17 scan (cve, pkg) pairs vs 17 exception pairs — empty set difference in both directions
Every entry scoped 12/12 carry purls, id, statement, expired_at

The two rescans are deliberately separate runs. A single run with --ignorefile cannot distinguish "correctly suppressed" from "wrote a file Trivy ignored".

The rescan matched the projected residual set exactly — 4 CRITICAL and 13 HIGH across 12 distinct CVEs, no finding added or dropped — so the ticket's residual table needed no correction.

Residual set (all excepted)

Package(s) CVE Sev Why unreachable
perl-base CVE-2026-13221, CVE-2026-42496, CVE-2026-57433, CVE-2026-8376 CRITICAL ×4 Nothing in the image executes perl
perl-base CVE-2026-42497, CVE-2026-48962, CVE-2026-57432, CVE-2026-9538 HIGH ×4 Same
libssl3t64, openssl, openssl-provider-legacy CVE-2026-14456 HIGH ×3 QUIC server DoS; the three API processes speak HTTP/1.1 over loopback behind PMM's nginx
libncursesw6, libtinfo6, ncurses-base, ncurses-bin CVE-2025-69720 HIGH ×4 Local vector needing user interaction with a crafted terminfo entry inside the container
gzip CVE-2026-41992 HIGH In-container archive paths use Python's gzip/tarfile stdlib, which link zlib. All four /usr/bin/tar -czf call sites run on a managed target host, not in this image
libacl1 CVE-2026-54369 HIGH Local privesc needing pre-existing privileges; the image runs as the unprivileged sep user

Four CRITICALs ship as documented exceptions, so the Percona CVE Process § 7 sign-off (engineering leadership, Product, and security) is requested on the ticket and cross-posted to PMM-15297.

Backwards compatibility

No public contract changes — no API response models, DB schemas, enums, config keys, error responses, seed data, or task payloads.

One behavioural change, confined to a build mode that ships nothing: an unrestricted (SEP_RESTRICT_APPS=0) build of this recipe would ship the report app without its system libraries, and app/sep/apps/report/service.py:37 imports weasyprint at module scope, so it would fail at load. This is not a regression to any shipped artifact — the app-restricted image is the only published one, and both make image and .github/workflows/ci.yml pass SEP_RESTRICT_APPS=1. The comment added beside the surviving apt-get layer names the five libraries and the condition under which they must return, which is what makes a future unrestricted build recoverable rather than a puzzle. weasyprint stays in pyproject.toml and in the wheel set; it contributes no findings, only its system libraries did.

Known limitations

  • Nothing in this repository consumes .trivyignore.yaml yet. No tracked workflow or pipeline invokes Trivy at all, so until a scan gate ships, this file is validated only by the gated rescan above, on the implementer's machine — a later edit could merge with nothing parsing it. Adding a .trivyignore.yaml row to .github/workflows/ci.yml's changes filter would not close this: the job that row triggers is build, which does static image inspection and runs no scanner. The recurring gate needs its own ticket — a workflow, a Trivy version pin, a failure threshold, the path filters, and the --ignorefile .trivyignore.yaml flag PMM's own scan does not carry — plus a revisit before expired_at: 2027-02-19 starts failing it.

Out of scope

Removing weasyprint from the Python dependency set; rebuilding or republishing released image tags; force-removing Debian-Essential packages (measured at 12 fewer findings and four fewer CRITICALs, and rejected on certainty grounds — --force-remove-essential exists precisely because dpkg cannot verify the outcome, and debconf is itself a perl program, so any later apt/dpkg operation in the image would break).

Tested

The side-car does not run standalone: SECRET_KEY has no default, and the three migration one-shots block on nc -z $SEP_DB_HOST $SEP_DB_PORT, whose host defaults to pmm-server. The scenarios below therefore need one PostgreSQL container plus the image on a shared network.

  • Build both stages from a clean cache: make builder, then RELEASE_VER=HEAD make image. Confirm both exit 0 — the runtime stage now has no compiler, so a wheel needing one at install time would fail here.
  • Inspect the rebuilt image and confirm g++ is absent, that linux-libc-dev, libglib2.0-0t64 and the five WeasyPrint libraries report not-installed, and that /etc/debian_version is 13.6 with util-linux 2.41.5-0+deb13u1.
  • Start postgres:16-alpine and sep:HEAD on a shared container network with SECRET_KEY, SEP_DB_HOST and SEP_DB_PASSWORD set, then exec the image's bundled healthcheck (sidecar/healthcheck.sh, installed at $APP_HOME/healthcheck.sh) inside the running container and confirm it prints healthy and exits 0. This asserts every non-one-shot supervisord program RUNNING, all three migrations complete, all three /health endpoints answering, and the broker responding to PING.
  • Confirm the shipped app set is unchanged: CONTAINER_RUNTIME=podman sidecar/verify_image_apps.sh sep:HEAD restricted.
  • Scan the rebuilt image with Trivy v0.69.3 at --severity HIGH,CRITICAL with no ignorefile, and confirm 17 findings, all in the Debian layer, with zero python-package findings.
  • Re-scan the same image adding --ignorefile .trivyignore.yaml --exit-code 1 and confirm it exits 0. Running both is what distinguishes a correctly suppressed set from a file Trivy never read.

Checklist

  • New/modified functions have type hints and rST docstrings (N/A for this change — no Python touched)
  • New tests added for new features or bug fixes (N/A for this change — a container-build recipe plus two non-Python files; verification is live evidence against the rebuilt image)
  • Database migrations generated if models changed (N/A for this change — no models changed)
  • User-facing changes documented (README, inline help, UI text) (N/A for this change)
  • Configuration changes documented with examples (N/A for this change)
  • Changelog fragment added under changelog.d/

Drop the runtime stage's first apt-get block: g++ pulled 43 transitive
packages including linux-libc-dev, and the WeasyPrint system stack backed
an app the embedded profile does not activate. Raise the base pin to
python:3.11.16-slim (Debian 13.6) and pair it with apt-get upgrade, since
the tag still ships util-linux 2.41-5. Drop the no-op pip install wheel —
the bumped base already supplies wheel 0.46.3.

Together these clear 84 of the 101 HIGH/CRITICAL findings. The remaining
17 ship as purl-scoped exceptions in .trivyignore.yaml, each carrying a
reachability argument and a six-month expiry.
@yyyyyyyan
yyyyyyyan requested review from a team as code owners August 19, 2026 15:57
Copilot AI balanced review requested due to automatic review settings August 19, 2026 15:57
@yyyyyyyan yyyyyyyan added the qa in progress Someone is currently testing this PR - do not merge it label Aug 19, 2026
@yyyyyyyan yyyyyyyan self-assigned this Aug 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the side-car image to remediate container vulnerabilities and document residual exceptions. The gzip exception’s call-site inventory is incomplete and needs correction.

Changes:

  • Refreshes the Python/Debian base and upgrades system packages.
  • Removes unused compiler and PDF-rendering dependencies.
  • Adds scoped Trivy exceptions and a security changelog fragment.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
sidecar/Containerfile.sidecar Hardens and reduces the runtime image.
.trivyignore.yaml Documents residual vulnerability exceptions.
changelog.d/SEP-1864.security.md Records the security remediation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .trivyignore.yaml
@yyyyyyyan yyyyyyyan added qa not required Merge without a QA sign-off: substitutes for 'qa passed' in label-gate. Does not skip any test job. and removed qa in progress Someone is currently testing this PR - do not merge it labels Aug 19, 2026
@yyyyyyyan
yyyyyyyan merged commit 2f4b91c into main Aug 19, 2026
23 of 33 checks passed
@yyyyyyyan
yyyyyyyan deleted the SEP-1864 branch August 19, 2026 16:12
@yyyyyyyan

Copy link
Copy Markdown
Contributor Author

Tracked the follow-up from the PR description as SEP-1869: Nothing in this repository consumes .trivyignore.yaml yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

qa not required Merge without a QA sign-off: substitutes for 'qa passed' in label-gate. Does not skip any test job.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants